Landscape Analysis for Shallow Neural Networks: Complete Classification of Critical Points for Affine Target Functions

Abstract In this paper, we analyze the landscape of true loss neural networks with one hidden layer and ReLU, leaky or quadratic activation. all three cases, provide a complete classification critical points in case where target function is affine one-dimensional. particular, show that there exist no local maxima clarify structure saddle points. Moreover, prove non-global minima can only be caused by ‘dead’ ReLU neurons. they do not appear Our approach combinatorial nature builds on careful analysis different types neurons occur.